Checkerboard artifact free sub-pixel convolution: A note on sub-pixel convolution, resize convolution and convolution resize

Convolutional neural networks (CNNs) are a popular and highly performant choice for pixel-wise dense prediction or generation. One of the commonly required components in such CNNs is a way to increase the resolution of the network’s input. The lower resolution inputs can be, for example, low-dimensional noise vectors in image generation [7] or low resolution (LR) feature maps for network visualization [4]. Originally described in Zeiler et al. [3], a network layer performing this upscaling task is commonly referred to as a “Deconvolution layer”, and has been used in a wide range of applications including super-resolution [1], semantic segmentation [5], flow estimation [6] and generative modeling [7]. The deconvolution layer can be described and implemented in various ways. This led to many names that are often used synonymously, including sub-pixel or fractional convolutional layer [7], transposed convolutional layer [8,9], inverse, up, backward convolutional layer [5,6] and more.